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Executive Summary 

During the first year of 
implementation, the emphasis of the 
America's Choice school reform design 
is an intensive focus on building 
students' writing skills. Writers 
workshop, the primary instructional 
emphasis of America's Choice during 
this year, is the component of the design 
for which teachers first receive in-depth 
training. 

In keeping with the emphasis of 
America's Choice, this year-one external 
evaluation study of the impact of 
America's Choice on student 
performance in Georgia focuses on 
student writing performance. The study 
examines changes in student writing 
performance from 2001 to 2002, the 
initial year of implementation of 
America's Choice in 109 Georgia 
elementary schools and 50 Georgia 
middle schools. Because state writing 
assessments were administered to 
students in fifth and eighth grades, our 
analyses are restricted to these grade 
levels. 

Overall, we found that students in 
America's Choice schools performed 
better on the state writing test than did 
students from similar Georgia schools. 
We found significantly greater gains in 
the writing performance of America's 
Choice schools in comparison to other 
Georgia schools, after adjusting for 
differences in prior school performance 
and a variety of school-level 
demographic characteristics. These 
effects were apparent in both the fifth- 
and eighth-grade results. More 
specifically, the average Georgia's 



Choice' elementary school had 20% 
more of its fifth-grade students scoring 
at the two highest levels of writing in 
2001 than in 2002. Similar Georgia's 
Choice schools had a significantly lower 
increase of 17%. For eighth grade, the 
average Georgia's Choice middle school 
had 29% more of its eighth-grade 
students scoring on target or better 
writing in 2001 than in 2002. Similar 
Georgia's Choice schools had a 
significantly lower increase of 25%. This 
suggests that while there was a sizeable 
improvement in writing performance 
statewide in Georgia, the improvements 
for Georgia's Choice schools were even 
larger than the trends for similar schools 
throughout the state. 

This study also contains exploratory 
results of the relationships between 
America's Choice school-level 
implementation measures and student 
learning. These measures are designed 
to assess schools' implementation on a 
variety of dimensions including 
understanding of the design, school and 
classroom implementation components, 
data use, parental involvement, and 
leadership. We found that none of these 
implementation indicators were reliable 
predictors of student achievement. Only 
in eighth grade did one of these 
implementation indicators — the use of 
data for planning and instruction — 
have a statistically significant 
relationship with gains in student test 
performance. This finding provides 
evidence in support of the substantial 
emphasis on ongoing student 
assessment that is part of the America's 
Choice design. However, our inability to 
detect relationships between 
achievement gains and other 
implementation indicators may be due 



' In Georgia, the America's Choice 
comprehensive school reform design was 
renamed Georgia's Choice. 
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to invariance in, or inaccuracy of, the 
measures. Therefore, while these results 
do not suggest that any particular 
component of the design is 
unimportant, they do provide evidence 
of sizeable program effects and they 
identify a particularly powerful 
component of the design related to these 
effects. 
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Introduction 

In this study, we investigate the 
impact of America's Choice on student 
writing performance in Georgia. The 
analyses in this study focus on the 
change that occurred during the first 
year of implementation, the 2001-2002 
school year. Two research questions 
guided these analyses: 

• What effect did Georgia's Ghoice 
have on the writing scores from the 
state assessment? 

• To what extent is the effect of 
Georgia's Ghoice related to 
implementation of specific 
components of the design? 

The data sources for this study were 
individual-level student test scores on 
the Georgia writing assessments, school- 
context measures (e.g., poverty rate, 
ethnic composition, etc.), and 
implementation data collected via a 
survey of cluster leaders, school 
principals, and the school leadership 
teams. 

This report is organized into six 
sections. In the next section, we present 
some background on Georgia's Choice 
and the rollout of the design during the 
first year. Then we describe our data 
sources and sample. Next, we describe 
our methods, and then we present the 
results of our analyses. Finally, we offer 
some interpretations of the results and 
explore the implications of these results 
for Georgia schools, America's Choice, 
and for comprehensive school reform in 
general. 



Background 

In the spring of 2001, the Georgia 
State Board of Education contracted 
with the National Center on Education 
and the Economy (NCEE) to provide 
America's Choice to approximately 160 
struggling elementary and middle 
schools throughout Georgia. This large- 
scale implementation was coordinated 
by both NCEE and the Georgia 
Department of Education (GA-DOE) 
and renamed Georgia's Choice. At the 
end of the first year, 109 elementary 
schools and 50 middle schools had 
implemented the first stages of the 
design. 

The rollout of America's Choice in 
Georgia focused on implementation of 
the writing and skills blocks during the 
first year. Implementation of the reading 
components did not begin until the end 
of the school year at the earliest, well 
after the state test had been 
administered. Math components of 
America's Choice were not 
implemented in the first year for 
elementary schools, and only two math 
core assignments were used during the 
first year in middle schools. These two 
core assignments were not used in pre- 
algebra or algebra classes. 

Implementation of the writing and 
skills components of America's Choice 
in elementary schools began in third 
and fourth grades, then expanded to 
include second and fifth grades, then 
proceeded to first and sixth grades. In 
middle schools, implementation of the 
writing and skills components of 
America's Choice began in eighth grade 
and expanded to seventh, then sixth 
grades. Because Georgia's Choice 
focused so specifically on writing in the 
first year, we restrict this year's study of 
impact to student writing scores. 
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Further analyses by CPRE in coming 
years will examine the impact of the 
design on student reading and 
mathematics scores in Georgia as those 
components of the design are 
implemented. 

Data and Sample 

Three sources of data were used in 
these analyses. Student scores on the 
Georgia writing assessment for all 
students in fifth and eighth grades 
during the 2000-2001 and 2001-2002 
school years were obtained from state 
databases. School-level variables 
providing information about the 
educational context of each school in the 
state was collected from the GA-DOE 
website. Implementation of the 
America's Choice design was measured 
through a survey administered by 
NCEE to Georgia's Choice cluster 
leaders, principals, and school 
leadership teams. 

Test Data 

Student writing performance was 
represented by student-level scores on 
state writing assessments. Table 1 shows 
the number of students taking the 
writing tests in grades 5 and 8 the year 



before Georgia's Choice began, and after 
one year of implementation. 

The GA-DOE descriptions of the 
grade 5 writing assessment and the 
middle grades writing assessment 
(eighth grade) are as follows. 

The writing assessment for grade 5 
consists of an evaluation of student 
response to an assigned prompt. 
Students are assigned a topic in one of 
two genres, imaginative story or 
personal narrative. Papers are scored by 
trained raters using a standardized 
scoring system. The papers are 
evaluated to determine the 
developmental stage that the writing 
represents. There are six developmental 
stages. 

Stage 1: The Emerging Writer 

• Little or no topic development, 
organization, and/ or detail. 

• Little awareness of audience or 
writing task. 

• Errors in surface features prevent 
the reader from understanding the 
writer's message. 



Table 1. Number of Students Tested in Writing 





Fifth Grade 


Eighth Grade 




Prior to 

Georgia’s Choice 


One Year into 
Georgia’s Choice 


Prior to 

Georgia’s Choice 


One Year into 
Georgia’s Choice 


Georgia’s Choice 
Schoois 


9,668 


9,448 


9,371 


9,627 


Other Schoois 


100,735 


100,247 


87,998 


89,535 
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Stage 2: The Developing Writer 

• Topic beginning to be developed. 
Response contains the beginning of 
an organization plan. 

• Limited awareness of audience 
and/ or task. 

• Simple word choice and sentence 
patterns. 

• Errors in surface features interfere 
with communication. 

Stage 3: The Focusing Writer 

• Topic clear even though 
development is incomplete. Plan 
apparent although ideas are loosely 
organized. 

• Sense of audience and/ or task. 

• Minimal variety of vocabulary and 
sentence patterns. 

• Errors in surface features interrupt 
the flow of communication. 

Stage 4: The Experimenting Writer 

• Topic clear and developed 
(development may be uneven). 
Clear plan with beginning, middle, 
and end (beginning and/ or ending 
may be clumsy). 

• Written for an audience. 

• Experiments with language and 
sentence patterns. Word 
combinations and word choice may 
be novel. 

• Errors in surface features may 
interrupt the flow of 
communication. 



Stage 5: The Engaging Writer 

• Topic well developed. Clear 
beginning, middle, and end. 
Organization sustains the writer's 
purpose. 

• Engages the reader. 

• Effective use of varied language and 
sentence patterns. 

• Errors in surface features do not 
interfere with meaning. 

Stage 6: The Extending Writer 

• Topic fully elaborated with rich 
details. Organization sustains the 
writer's purpose and moves the 
reader through the piece. 

• Engages and sustains the reader's 
interest. 

• Creative and novel use of language 
and effective use of varied sentence 
patterns. 

• Errors in surface features do not 
interfere with meaning. 

The Georgia middle grades writing 
assessment is an evaluation of eighth- 
grade students' written responses to an 
assigned topic. In responding to the 
assigned topic within a two-hour timed 
period, the students are given the choice 
of writing a narrative story, an 
expository report, or a persuasive essay. 
The student responses are read by 
trained raters and evaluated in the areas 
of content/ organization, style, sentence 
formation, usage, and mechanics. The 
students receive feedback in the form of 
a scale score ranging from 300 to 400, 
performance scores in each of the areas 
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Table 2. Means and Standard Deviations of School-Context Variables 
for Georgia’s Choice and Other Schools 





Georgia’s Choice 
Elementary 
Schools 
(N=109) 


Other 

Elementary Schools 
(N=963) 


Georgia’s 
Choice Middle 
Schools 
(N=50) 


Other 

Middle Schools 
(N=356) 


School Size 


528 


597 


722 


836 




(221) 


(238) 


(295) 


(363) 


Percent Minority 


79 


48 


77 


41 




(27) 


(32) 


(23) 


(28) 


Percent Retained 


3 


3 


5 


3 


in Grade 


(3) 


(3) 


(3) 


(3) 


Percent Free/Reduced- 


87 


53 


75 


46 


Price Lunch Eligibie 


(15) 


(28) 


(12) 


(23) 


Percent Limited Engiish 


4 


3 


1 


2 


Proficient 


(8) 


(6) 


(4) 


(3) 


Percent Speciai 


12 


12 


13 


13 


Education 


(5) 


(4) 


(4) 


(4) 


Average Teacher Saiary 


42,053 


43,543 


42,050 


43,389 




(2,421) 


(2,686) 


(2,328) 


(2,579) 


Average Teacher Years 


12 


13 


13 


12 


of Experience 


(2) 


(3) 


(3) 


(2) 


Pupii/Teacher Ratio 


17 


16 


16 


16 




(3) 


(3) 


(2) 


(2) 


Average Fifth-Grade 


3.9 


4.2 






Writing Levei for 


(0.2) 


(0.4) 






2000-2001 










Percent Meeting or 






72 


84 


Exceeding the Eighth- 






(9) 


(9) 


Grade Writing Standard 










for 2000-2001 











evaluated, and total performance level 
of not on target, on target, and exceeds 
target. Only the performance-level 
scores were available for individual 
students. The scale scores are not used 
in this study. 

School-Context Data 

School-level variables providing 
information about the educational 



context of each school included 
information about the demographic 
composition of the student body, 
teacher characteristics, and the previous 
writing performance of students in each 
school. Table 2 shows the means and 
standard deviations for these school- 
level variables for Georgia's Choice and 
other schools for the 2001-2002 school 
year, along with the writing scores for 
the 2000-2001 school year. 
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On average, the Georgia's Choice 
schools are very similar to the 
comparison schools on 9 of the 11 
school-context variables. The only 
practical differences are that the 
Georgia's Choice schools have higher 
concentrations of minority students, and 
they also have higher concentrations of 
students from poor families. There is 
also a slight difference in the average 
teacher salary. Teachers in the 
comparisons schools are paid about 
3.5% more than teachers in the 
Georgia's Choice schools. 

Implementation Data 

In an effort to measure the 
implementation of the America's Choice 
design, NCEE administered a survey, 
called the Diagnostic and Assessment 
Tool (DAT), to Georgia's Choice schools. 
Responses to implementation items 
were on a five-point scale ranging from 
not in place to fully implemented. 
Responses to other items were on a five- 
point scale ranging from strongly 
disagree to strongly agree. The responses 
to the survey for each school were 
arrived at through a consensus decision 
by cluster leaders, principals, and school 
leadership teams. 

CERE conducted factor analyses of 
the DAT data and derived seven scales 
from the survey responses. These were: 

• Understanding of the America's 
Choice design, 

• Implementation of school-level 
structural elements, 

• Implementation of classroom-level 
structural elements, 

• Use of data for planning and 
instruction. 



• Parent and community involvement, 

• Commitment to the design, and 

• Active and effective leadership. 

The items comprising these scales 
and estimates of the internal consistency 
(i.e., Cronbach's Alpha) for each of the 
seven scales are shown in Appendix A. 
Eor elementary schools, these 
reliabilities range from .69 to .88. Eor 
middle schools, these reliabilities range 
from .68 to .86. 

Methods 

In accordance with the two research 
questions stated in the introduction of 
this report, the analyses for this study 
proceeded in two stages. The first set of 
analyses compared the gains in writing 
scores from 2001 to 2002 for Georgia's 
Choice schools to other schools after 
controlling for prior performance and 
other school-level variables. The second 
set of analyses included only the 
Georgia's Choice schools and explored 
possible links between quality of 
implementation and outcomes. Both sets 
of analyses used the same analytic 
approach and similar statistical models 
for predicting gains in school-level 
performance from 2001 to 2002. 

Ideally, an analysis of achievement 
gains would model change at the 
individual level (i.e., the student level). 
In order to do this, every student needs 
to be tested at least twice, before and 
after the implementation of the 
program. Unfortunately, the Georgia 
writing test was administered to fifth, 
eighth, and eleventh graders only. With 
no writing test in fourth or seventh 
grades, there is no available information 
about the writing performance of fifth- 
and eighth-grade students at the end of 
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the previous year (i.e., when they were 
in fourth or seventh grade). In fact, there 
was no statewide assessment in any 
subject administered to seventh graders 
during the 2000-2001 school year. There 
was an assessment administered to 
fourth graders called the Criterion 
Referenced Competency Test, which 
includes reading and English /language 
arts components. Therefore, a reading or 
language arts score could be used as a 
control variable for the fifth-grade 
writing scores. Unfortunately, 
difficulties in obtaining data that 
included consistent student identifiers 
across years prohibited this type of 
analysis. 

Fortunately, we were able to 
develop an alternative analytical 
approach that is able to model school- 
level changes in writing scores using 
individual students' writing scores and 
school-level variables. This approach 
enabled us to compare changes in 
writing scores at the school level, while 
controlling for differences in the 
characteristics of the student 
populations in the schools along with 
other school characteristics. More 
importantly, because the dependent 
variable in these analyses is the 
individual student score, there was no 
loss of precision usually associated with 
the use of school-level aggregates of 
student-level scores (e.g., school average 
scores, percent proficient, etc.). In 
essence, the model we developed 
compares sequential cohorts of students 
within schools, and the impact estimates 
produced show effects at the school 
level.^ 



^ Although the analysis of individual change is 
often preferred for impact analyses, an analysis 
of sequential cohorts is appropriate here given 
that the impact estimate pertains to change at the 
school level, and the sequential cohorts within 
each school are nearly identical in terms of size, 
ethnic composition, and poverty rates. 



The student-level achievement 
scores in these data were nested within 
schools, so a two-level hierarchical 
model (Raudenbush & Bryk, 2002) was 
used in these analyses. Furthermore, 
because the achievement scores were 
coded as ordered categories of 
proficiency, an ordinal logistic 
regression model was used to model 
differences in achievement scores. In 
order to model change in writing scores 
from 2001 to 2002 for a particular grade 
level, all students tested in that grade in 
both years were included in the 
analysis. An indicator variable 
signifying the year in which the student 
was tested was included as a random 
effect at Fevel 1 (coded 1 for 2002 and 0 
for 2001). At Fevel 2, the school-context 
variables are each centered around their 
grand means (see Raudenbush & Bryk, 
2002, pp. 31-35 for a discussion of 
centering and its implications). 
Adapting the notation of Raudenbush 
and Bryk (2002) with some 
modifications that simplify the 
formulas, the general analytic model 
had the mathematical form shown on 
page 7. 

On the left side of the equation for 
Fevel 1 is the formula for the log-odds 
for observing a score greater than or 
equal to m. This is equal to the natural 
logarithm of the probability of 
observing a score greater than or equal 
to m divided by the probability of 
observing a score less than m. The Fevel 
1 intercept (i.e., Po,) shows the overall 
baseline achievement for school j (i.e., 
before the implementation of Georgia's 
Choice). The Fevel 1 slope (i.e., Pij) 
shows the change in the overall baseline 
achievement for school j during the first 
year of Georgia's Choice. The final term 
shows the difference in the log-odds of 





The Impact of America’s Choice on Writing Performance in Georgia: First-Year Resuits 



7 



Level 1: 



, ( Prob(K. >m)' 
In 

^Prob(}^. <m) J 

Level 2: 






Ay = 7oo + r qXj. + 7oi {Georgia' s Choice) + u^j 

Ay = /lo + 7 n (2001 school average writing score) y +rjXj + /^^{Georgia' sChoice) + u^j 



observing scores in M-2 adjacent 
categories for school j. Note that the 
probability of observing a score of 1 or 
higher is 100% (i.e., Prob(L, 7 ^ 1 )=l); 
therefore, this model treats writing 
scores equal to 1 as the reference 
category, and 85 is set equal to 0 to 
define the model. 

In the first equation at Level 2, the 
overall baseline achievement for the J 
schools (i.e., fioj) is modeled as an 
average baseline achievement for non- 
Georgia's Choice schools (i.e., 700 ), plus 
a vector of fixed effects (i.e., LoXj) for the 
context variables of school j, plus an 
effect of Georgia's Choice status (i.e., 
7oi), plus a residual term (i.e., u oy) for 
school j. 

In the second equation at Level 2, 
the change in overall achievement for 
the / schools (i.e., ydyy) is modeled as an 
average change for non-Georgia's 
Choice schools (i.e., 710 ), plus an effect 
of initial school achievement (i.e., 711), 
plus a vector of fixed effects (i.e., L iXj) 
for the context variables of school j, plus 
an effect of Georgia's Choice status (i.e., 
712), plus a residual term (i.e., u y) for 
school j. 

In the first stage of analyses, the 
indicator variable representing each 
school's participation status in Georgia's 



Choice included in both Level 2 
equations produced estimates of the 
differences in baseline achievement 
between Georgia's Choice schools and 
other schools before the implementation 
of Georgia's Choice (i.e., 701 ), and the 
differences between Georgia's Choice 
schools and other schools in the change 
in achievement from 2001 to 2002 (i.e., 

712)- 

Effect sizes from these models 
represent differences in log-odds. The 
effects are usually exponentiated to 
produce odds ratios. While this is 
appropriate for the estimate of baseline 
differences, it is not helpful for 
interpreting the differences in gains (i.e., 
exponentiation would produce a ratio of 
odds ratios). Therefore, to aid in 
interpretation of these effect estimates, 
the estimates from the model are used to 
produce expected proportions of 
students scoring at each proficiency 
level for 2000-2001 and 2001-2002. These 
are much easier to interpret, and they 
make the magnitude of the effect more 
evident. 

In the second stage of the analysis, 
factor scores for the seven 
implementation scales were included as 
fixed effects predicting change in 
achievement from 2001 to 2002 for the 
Georgia's Choice schools. Due to high 
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correlations between the 
implementation scales and the potential 
for collinearity, we decided to run 
separate models for each of the seven 
implementation variables. These models 
included the same control variables as 
in the first stage, along with a single 
implementation variable predicting 
differences in achievement gains only, 
not differences in initial status. Any 
implementation variables found to have 
significant relationships with writing 
score gains were then reanalyzed after 
controlling for other implementation 
factors. This was done to determine the 
extent to which significant relationships 
were due to general implementation 
quality or to the quality of 
implementation for specific components 
of Georgia's Choice. 

All models were estimated via 
penalized quasi-likelihood as 
implemented in HLM version 5.05 
(Raudenbush, Bryk, & Congdon, 2001). 

Results 

In this section, we present the results 
from both stages of analyses. First, we 
report the results of our impact analysis, 
addressing the first research question, 
which focuses on the gains in student 
writing scores associated with 
participation in Georgia's Choice. Then 
we report the results of our 
implementation analyses, addressing 
the second research question, which 
focuses on possible links between 
variation in the quality of 
implementation of Georgia's Choice and 
variation in the gains experienced by 
Georgia's Choice schools. 



Impact of Georgia’s Choice on 
Writing Scores 

Estimates of the impact of Georgia's 
Choice on writing scores were produced 
separately for fifth and eighth grades. 
This was done by including in both 
Level 2 equations of the general analytic 
model an indicator variable coded 1 for 
Georgia's Choice schools and 0 for 
comparison schools. Table 3 shows the 
estimates of baseline differences and 
program impact (i.e., gains) for 
Georgia's Choice schools for the fifth 
and eighth grades. The full models are 
shown in Appendix B.^ 

For the fifth-grade model, these 
estimates show that, after controlling for 
several school characteristics, the 
baseline writing scores of Georgia's 
Choice elementary schools (i.e., before 
implementation) were not significantly 
different from non-Georgia's Choice 
schools, and the gains in writing scores 
for Georgia's Choice elementary schools 
after one year of implementation were 
significantly greater than the gains 
experienced by non-Georgia's Choice 
schools throughout the state. More 
specifically, during the 2001-2002 school 
year, Georgia's Choice elementary 
schools experienced significantly greater 
increases in the proportion of fifth 
graders scoring in higher writing 

’ The robustness of these effect estimates to 
model misspecification (e.g., nonlinearities, 
omitted interactions) was evaluated by running 
each model using subsets of the full sample after 
stratifying on prior achievement, ethnic 
composition, and poverty rates. Estimates of the 
effects of Georgia's Choice on test score 
improvements were nearly identical to those 
from the full models for all models run for both 
fifth and eighth grades. The estimates were 
statistically significant for all four models run for 
fifth grade, and significant for two out of four 
models run for eighth grade. Note that the 
sample of schools is much smaller at eighth 
grade. 



The Impact of America’s Choice on Writing Performance in Georgia: First-Year Resuits 



9 



Table 3. Results of the Regression Model Predicting Baseline (2001) Writing Scores 



and Change in Writing Scores (2001 to 2002) for Fifth and Eighth Grades 




Regression 

Coefficient 


Standard 

Error 


Difference in Fifth-Grade Writing Scores Prior to Georgia’s 
Choice 


-0.03 


0.05 


Difference in 2001-2002 Gains for Fifth-Grade Writing Scores 


0.16* 


0.06 


Difference in Eighth-Grade Writing Scores Prior to Georgia’s 
Choice 


-0.17* 


0.08 


Difference in 2001-2002 Gains for Eighth-Grade Writing 
Scores 


0.17** 


0.06 



*p<.05, **p<.01 



proficiency categories than similar 
schools not participating in Georgia's 
Choice. 

For the eighth-grade model, these 
estimates show that, after controlling for 
several school characteristics, the 
writing scores of Georgia's Choice 
middle schools for the 2000-2001 school 
year (i.e., before implementation) were 
significantly lower than those from non- 
Georgia's Choice schools, and the gains 
in writing scores for Georgia's Choice 
middle schools after one year of 
implementation were significantly 
greater than non-Georgia's Choice 
schools. More specifically, during the 
2001-2002 school year, Georgia's Choice 
middle schools experienced significantly 



greater increases in the proportion of 
students scoring in higher writing 
proficiency categories than similar 
schools not participating in Georgia's 
Choice. 

The magnitude of these effect 
estimates may be more easily 
understood when represented in terms 
of the expected proportions of students 
scoring at each proficiency level in the 
baseline year and also after one year of 
implementation of Georgia's Choice. 
Table 4 shows these proportions for the 
fifth grade as predicted by the statistical 
analysis. Whereas the proportion of 
students at each proficiency category is 



Table 4. Predicted Proportions of Fifth Graders at Various Proficiency Levels 
Before and After the First Year of Georgia’s Choice 



Writing Proficiency Levei 



1 


2 


3 


4 


5 


6 



Average of Georgia’s Choice Schoois 


(before impiementation) 2000-2001 


2 


7 


36 


41 


12 


2 


(after one year) 2001-2002 


1 


3 


18 


45 


27 


7 


Average of Simiiar Comparison Schoois 


2000-2001 


2 


7 


35 


41 


12 


2 


2001-2002 


1 


3 


19 


46 


25 


6 



The Impact of America’s Choice on Writing Performance in Georgia: First-Year Resuits 



10 



Table 5. Predicted Proportions of Eighth Graders at Various Proficiency Levels 
Before and After the First Year of Georgia’s Choice 



Writing Proficiency Levei 





1 


2 


3 


Average of Georgia’s Choice Schoois 


(before impiementation) 2000-2001 


52 


47 


2 


(after one year) 2001-2002 


22 


73 


5 


Average of Simiiar Comparison Schoois 


2000-2001 


47 


51 


2 


2001-2002 


22 


73 


5 



nearly the same for the baseline year 
(2000-2001) for both Georgia's Choice 
schools and comparison schools with 
similar characteristics, the Georgia's 
Choice schools have 20% more students 
at levels 5 and 6 after one year 
compared to an increase of 17% for 
similar Georgia schools. 

Table 5 shows these proportions for 
the eighth grade as predicted by the 
statistical analysis. The average 
Georgia's Choice middle school had 
29% more of its eighth-grade students 
scoring on target or better writing in 
2001 than in 2002. Similar Georgia's 
Choice schools had a significantly lower 
increase of 25%. In other words, 
whereas the proportion of students 
scoring proficient (i.e., levels 2 and 3) is 
lower in Georgia's Choice schools 
during the baseline year (2000-2001), the 
Georgia's Choice schools have caught 
up to the comparison schools with 
similar characteristics after one year of 
implementation. 

This suggests that while there was a 
sizeable improvement in writing 
performance statewide in Georgia, the 
improvements for Georgia's Choice 
schools were even larger than the trends 
for similar schools throughout the state. 



Relationship Between Quality of 
Implementation and Impact of 
Georgia’s Choice 

Estimates of the degree to which the 
quality of implementation in Georgia's 
Choice schools was related to their gains 
in fifth- and eighth-grade writing scores 
were also produced. The seven 
measures of implementation described 
earlier were used in these analyses. Each 
measure was scaled to have a mean of 0 
and a standard deviation of 1, so that 
regression coefficients show the change 
in odds per standard deviation in each 
implementation measure. These results 
are shown in Table 6 with the full 
models shown in Appendix C. 

Use of data for planning and 
instruction among middle schools was 
the only implementation measure 
shown to have a significant relationship 
with gains in writing scores. Georgia's 
Choice schools with a score of one 
standard deviation above the mean on 
the data-use scale had a 13% greater 
odds ratio (e '^=1.13) of students scoring 
in higher writing proficiency categories 
than Georgia's Choice schools with an 
average score on the data-use scale. This 



The Impact of America’s Choice on Writing Performance in Georgia: First-Year Resuits 



11 



Table 6. Results of the Regression Model Predicting Change in Writing Scores 
(2001 to 2002) for Fifth Grade from Seven Implementation Factors 





Regression 

Coefficient 


Standard 

Error 


Fifth Grade 


Understanding of the America’s Choice Design 


-.00 


.05 


Schooi-Levei Structurai Eiements 


.02 


.05 


Ciassroom-Levei Structurai Eiements 


.02 


.05 


Use of Data for Pianning and Instruction 


-.01 


.05 


Parent and Community Invoivement 


.01 


.05 


Commitment to the Design 


.00 


.05 


Active and Effective Leadership 


.06 


.05 


Eighth Grade 


Understanding of the America’s Choice Design 


-.02 


.07 


Schooi-Levei Structurai Eiements 


-.03 


.06 


Ciassroom-Levei Structurai Eiements 


-.04 


.08 


Use of Data for Pianning and Instruction 


.12* 


.05 


Parent and Community Invoivement 


.00 


.06 


Commitment to the Design 


.00 


.06 


Active and Effective Leadership 


.09 


.06 



*p<.05 



translates to 4% more students scoring 
on target or above for Georgia's Choice 
schools one standard deviation above 
the mean on use of data for planning 
and instruction, and 6% more students 
scoring on target or above for Georgia's 
Choice schools two standard deviations 
above the mean on use of data for 
planning and instruction. 

The relationship between data use 
for planning and instruction and gains 
in middle school writing scores 
remained significant after controlling for 
the six other implementation measures. 
In fact, the parameter estimate of the 
effect of this variable increased to .23 
(p<.01) after including these additional 
control variables. This suggests that, for 
middle schools in Georgia, data use was 
important as a distinct component of the 
America's Choice design, potentially 
driving a significant portion of the gains 
experienced by schools participating in 
Georgia's Choice. 



Conclusions and 
Implications 

The results of this study suggest two 
things. First, the America's Choice 
design as implemented in elementary 
and middle schools in Georgia 
produced significant gains on the state's 
writing assessments. Second, those 
middle schools participating in 
Georgia's Choice that had greater use of 
data for planning and instruction 
experienced greater gains in writing 
scores. These findings provide 
additional evidence that the America's 
Choice comprehensive school reform 
design has had a positive impact on the 
performance of students and schools 
across the nation. 

The effect sizes for this study are 
small relative to accepted standards in 
statistics (Cohen, 1988). However, it is 
generally the case that moderate or large 
effects are rare in large-scale education 
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research. For example, Borman, Hewes, 
Overman, and Brown (2003), in their 
meta-analysis of research on the impacts 
of comprehensive school reform, found 
that even the most effective designs 
produced only small effects over the 
course of one year. Fortunately, these 
effects cumulate over the course of time, 
and as Borman and Hewes (2002) 
demonstrate, these positive impacts can 
lead to considerable differences in the 
educational outcomes of many students. 

In previous studies conducted as 
part of CPRE's evaluation of America's 
Choice, it was usually the case that 
greater effects were seen in elementary 
grades, rather than in middle grades. 
This may not be the case in this study, 
simply due to the fact that the design 
was implemented longer in the eighth 
grade than in the fifth grade. As noted 
in this report, implementation of the 
design in elementary schools began in 
fourth grade and expanded to fifth 
grade. In middle schools, 
implementation began in the eighth 
grade, so the program was in place 
longer in eighth grade than in fifth 
grade. As such, we might begin to see 
larger effects for the elementary grades 
as implementation continues. 

It is interesting that the use of data 
for planning and instruction appeared 
as an important predictor of writing 
score gains among Georgia's Choice 
schools. The fact that this relationship 
grew even stronger when other 
measures of implementation were held 
constant suggests that data use is 
important as a distinct component of the 
design. In other words, data use does 
not come automatically when schools 
implement the design; however, when 
schools use data for planning and 
instruction, regardless of their 
implementation of other components. 



they experience greater gains. The 
support for this conclusion is somewhat 
fragile because a similar relationship 
was not found for fifth-grade writing 
scores. Nevertheless, these results are 
very interesting to say the least, and 
further research on schools' and 
teachers' use of data is surely a 
promising venture. 

The apparent lack of relationships 
between implementation measures and 
outcomes is also interesting. However, 
we are not surprised by this lack of 
relationships. In a previous study, we 
discovered that the majority of the 
variation in implementation of 
America's Choice occurred among 
teachers within schools, not between 
schools (Supovitz & May, in press). The 
lack of relationships between school- 
level implementation and outcomes 
may be due to a lack of variation in 
implementation at the school level. In 
other words, the variation in 
implementation among schools may be 
quite small relative to the variation in 
implementation among teachers. Given 
the absence of teacher-level data from 
this study, evaluation of this hypothesis 
must be left for future research. 

It is important to consider the 
limitations of this study when 
interpreting its results. Specifically, 
there are three limitations worth 
discussing here. First, this study is an 
observational quasi-experiment; it is not 
a randomized experiment. However, as 
Cook (2002) points out, while such 
quasi-experimental studies may be less 
efficient than randomized experiments, 
there is no evidence to suggest that they 
systematically underestimate or 
overestimate effects. A second limitation 
is related to the fact that student-level 
demographics were not included as 
control variables in these analyses. Yet, 
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given the differences between Georgia's 
Choice schools and comparison schools, 
inclusion of these student-level controls 
would most likely increase the effect 
estimates. Lastly, and most importantly, 
this study is limited by the fact that it 
does not model individual change. That 
is, each student in these analyses is 
tested only once, so there is no direct 
measure of the increase in writing skills 
for any particular student. Nevertheless, 
these analyses do model school-level 
change, and given that most states' 
accountability systems focus on school- 
level scores for particular grades, this 
type of change is important. The de 
facto assumption that accompanies such 
a school-level change analysis is that the 
differences between cohorts of students 
in the same school are relatively small. 

In other words, it is assumed that the 
performance of eighth graders in 2001 is 
a good indicator of the performance of 
eighth graders in 2002 in the absence of 
any intervention. Given the relative 
homogeneity of schools and the fact that 
the demographics of schools usually 
change very slowly (which is the case 
for these data), we believe this 
assumption is a reasonable one. 

In summary, the results of this study 
suggest that Georgia's Choice has had a 
positive impact on writing scores in the 
fifth and eighth grades. This is a very 
promising finding given that, to date, 
this is the largest scale-up of a 
comprehensive school reform model 
within a single site. 
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About the America’s 
Choice Design 

The America's Choice school design 
is a K- 12 comprehensive school reform 
model developed by the National 
Center on Education and the Economy 
(NCEE). America's Choice is a well- 
established school reform model 
currently being implemented in over 
500 schools across the nation. America's 
Choice focuses on raising academic 
achievement by providing a rigorous 
standards-based curriculum and safety 
nets for all students. A stated goal of the 
design is "to ensure that every student 
is successful on state and local 
assessments and prepared for college" 
(NCEE, n.d.). 

America's Choice does not offer 
schools a script or a paint-by-numbers 
approach to reformed instruction. 
America's Choice recognizes that the 
pace of change will vary from school to 
school and, thus, the model does not 
have a rigid implementation schedule. 
Rather, the core of the America's Choice 
design contains a set of principles about 
the purpose of schooling and how 
schools should operate, and it provides 
a set of tools for building a program 
based on those principles. These 
essential principles and tools include: 

• High expectations for all students, 
with communication of those 
expectations through explicit 
performance standards that are 
aligned to assessments and include 
examples of student work that meet 
the standards. These New Standards 
Performance Standards provide a 
common set of expectations for 
students and teachers. 



• The implementation of standards- 
based literacy, math, and skills 
blocks, which happen every day for 
every child, and dramatically 
change teaching and learning in 
every classroom. The rituals and 
routines associated with these blocks 
are designed to prepare students to 
deal with demanding content and 
become independent learners. 

• A common core curriculum that is 
aligned with the standards. Through 
the America's Choice literacy 
workshops, core assignments, and 
foundations of advanced 
mathematics, school life is organized 
around a core curriculum. These 
curricular components are designed 
to help students develop key skills, 
convey core concepts, and apply 
what they know. 

• Ongoing assessment of students in 
order to inform daily instruction 
through the use of standards-based 
assessments, including the New 
Standards Reference Examination, 
which are aligned with the 
standards and the core curriculum 
and provide detailed feedback to 
teachers and students about student 
skill levels in relation to the 
standards. 

• School-embedded, ongoing, teacher 
professional development led by a 
full-time literacy coach and designed 
to strengthen teachers' knowledge of 
the America's Choice approach to 
teaching and learning. This includes 
learning how to conduct a close 
analysis of their students' work in 
relation to standards, and using this 
knowledge to develop lessons 
calibrated to the needs of different 
students. 
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• "Safety nets," including tutoring and 
course recovery programs, that are 
structured into the school day and 
school year, and that provide 
students with extensive support and 
multiple opportunities to achieve 
the standards. 

• A school leadership team, led by the 
principal and subject-matter 
coaches, that coordinates 
implementation through a variety of 
means. These include setting 
performance targets and analyzing 
student work on a variety of 
measures, training teachers, 
adjusting school schedules, and 
implementing safety-net programs 
to provide time for students to 
receive additional instruction. 

In order to become an America's 
Choice school, over 80% of a school's 
faculty must indicate their commitment 
to the America's Choice design and 
agree to implement the program over 
three years. Each school must assign 
personnel as coaches to lead the 
implementation of the design, and a 
parent/ community outreach 
coordinator to ensure that students get 
needed support services. 

About CPRE’s Evaluation 
of America’s Choice 

The Consortium for Policy Research 
in Education (CPRE) at the University of 
Pennsylvania was contracted by NCEE 
in 1998 to conduct the external 
evaluation of the America's Choice 
school design. Each year, CPRE designs 
and conducts a series of targeted studies 
on the implementation and impacts of 
the America's Choice design. 



The publication of this report 
follows the release of several other 
studies by CPRE on the implementation 
and impact of America's Choice across 
the nation using both qualitative and 
quantitative methods. Those previous 
reports can be viewed as separate pieces 
or as complements to the information 
presented in this report. 

The purpose of CPRE's evaluation is 
to provide formative feedback to NCEE 
and America's Choice schools about 
emerging trends in the implementation 
of the design, and to seek evidence of 
the impacts of the design using accepted 
high standards of evaluation design and 
analysis methodologies. 

CPRE's evaluation of America's 
Choice is guided by three overarching 
evaluation questions. Eirst, is America's 
Choice being carried out in the manner 
envisioned — that is, how are teachers 
and school administrators 
understanding and implementing the 
many facets of the America's Choice 
reform design? Second, as a result of 
their implementation of America's 
Choice, are the instructional practices of 
teachers changing in ways that would 
improve student learning? Third, do 
America's Choice schools experience 
improvements in student achievement, 
and to what degree can changes in 
student performance be attributed to the 
design? Within this framework, annual 
evaluation studies target specific aspects 
of the America's Choice design for more 
in-depth investigation. 

To address these questions, the 
CPRE evaluation team gathers a broad 
array of qualitative and quantitative 
data to develop a rich and valid picture 
of the implementation process over time 
and to capture the impacts of the design 
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on students and teachers. Data sources 
include: 

• Surveys of teachers and 
administrators in America's Choice 
schools nationwide. 

• Site visits to schools across the 
nation to observe classroom 
instruction, examine implementation 
artifacts, and interview teachers, 
students, and school administrators. 

• Telephone interviews with NCEE 
staff, school faculty members, and 
school and district administrators. 

• Document reviews. 

• Observations of national, regional, 
and school-level professional 
development. 

• Collection of student performance 
measures, including state and local 
tests, the New Standards Reference 
Examination, and more authentic 
samples of student work products. 

After data collection, CERE 
evaluation team members analyze the 
data using appropriate qualitative and 
quantitative research techniques in 
order to identify patterns of intended 
and unintended consequences and to 
detect effects of the design on students, 
teachers, and schools. The results are 
reported in a series of thematic 
evaluation reports that are released each 
year. 

Additional Reading on 
America’s Choice 

The following reports are currently 
available from CERE. Erint copies are 
available at no cost by emailing 



cpre@gse.upenn.edu, or by calling 215- 
573-0700. Copies can also be 
downloaded at www.cpre.org/ 
Research/Research_Eroject_America's_ 
Choice.htm. 

• A Longitudinal Study of the Impact 
of America's Choice on Student 
Eerformance in Rochester, New 
York, 1998-2003 (Henry May, 
Jonathan A. Supovitz, and David 
Eerda, July 2004) 

• Mapping a Course for Improved 
Student Learning: How Innovative 
Schools Systematically Use Student 
Eerformance Data to Guide 
Improvement (Jonathan A. Supovitz 
and Valerie Klein, November 2003) 

• Teacher and Coach Implementation 
of Writers Workshop in America's 
Choice Schools, 2001 and 2002 

(Amy J. Bach and Jonathan A. 
Supovitz, October 2003) 

• The Heart of the Matter: The 
Coaching Model in America's 
Choice Schools (Susan M. Eoglinco, 
Amy J. Bach, Kate Hovde, Sheila 
Rosenblum, Marisa Saunders, and 
Jonathan A. Supovitz, May 2003) 

• The Relationship Between Teacher 
Implementation of America's 
Choice and Student Learning in 
Elainfield, New Jersey (Jonathan A. 
Supovitz and Henry May, January 
2003) 

• Impact of America's Choice on 
Student Eerformance in Duval 
County, Florida (Jonathan A. 
Supovitz, Brooke Snyder Taylor, and 
Henry May, October 2002) 
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• Implementation of the America's 
Choice Literacy Workshops 

(Jonathan A. Supovitz, Susan M. 
Poglinco, and Amy J. Bach, April 
2002) 

• Instructional Leadership in a 
Standards-Based Reform (Jonathan 
A. Supovitz and Susan M. Poglinco, 
December 2001) 

• Moving Mountains: Successes and 
Challenges of the America's Choice 
Comprehensive School Reform 
Design (Jonathan A. Supovitz, 

Susan M. Poglinco, and Brooke 
Snyder, March 2001) 

• America's Choice Comprehensive 
School Reform Design: First-Year 
Implementation Evaluation 
Summary (Thomas Corcoran, 
Margaret Hoppe, Theresa Luhm, 
and Jonathan A. Supovitz, February 
2000) 
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Appendix A: Scales of Implementation 

Table Al. Factors, Items, and Estimates of Internal Consistency 
for the Elementary School Implementation Survey — Primary 

Cron bach’s 

Elementary Scales and Items Alpha 



Factor 1: Understanding of the America’s Choice design 0.8449 

Teachers and administrators understand the connections between state/district content 
standards and the New Standards Performance Standards. 

There is a high ievei of understanding of the America’s Choice design among staff in 
target primary ciassrooms. 

There is a high ievei of understanding of the America’s Choice design among aii staff at 
this schooi. 

There is a high ievei of understanding of the America’s Choice design among students. 

There is a high ievei of understanding of the America’s Choice design among 
parents/guardians and the iocai community. 

Factor 2: Implementation of school-level structural elements 0.7165 

The whole school community impiements the 25-Book Campaign. 

A Principai-sponsored Book-of-the-Month program is in piace. 

Exhibits of student work that approaches or meets the standards, aiong with appropriate 
standards and eiements, are dispiayed on buiietin boards in ciassrooms and haiis. 

Modei iiteracy ciassrooms are estabiished at the second and fourth grades and Skiiis 
Biock in a kindergarten/first-grade ciassroom. 

The Upper-Eiementary Coach impiements smali group and tutoriai programs for students 
needing assistance with print/sound code. 

Summer iiteracy catch-ups are pianned for students who are significantiy beiow standard 
in iiteracy. 

Time is scheduied for meetings of teams of grade-ievei and same-subject teachers. 

The master scheduie inciudes a 2 V 2 -hour Literacy Biock and a 1-hour Mathematics Biock, 
and time is scheduied for meetings of grade-ievei and same-subject teachers. 

Ciass teachers are identified who wiii teach the same students for the next 2-3 years and 
aiiocated to teaching teams for the commencement of the foiiowing schooi year. 

There are scheduied Study Group meetings of teachers to read and discuss monographs. 

There are scheduied Teacher Meetings, coordinated by the Coaches, focusing on 
ciassroom teaching and iearning. 
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Table Al. Factors, Items, and Estimates of Internal Consistency 



for the Elementary School Implementation Survey — Primary (continued) 




Elementary Scales and Items 


Cronbach’s 

Alpha 


Factor 3: Implementation of classroom-level structural elements 

Teachers in target classrooms (10 per coach) use the New Standards Performance 
Standards for English/language arts and the primary literacy standards to plan instruction 
and to develop lessons. 

Teachers in target classrooms implement one Genre Study at each grade level. 

Teachers in target primary classrooms (grades K-2) implement the Skill Development 
Block and the Writers and Readers Workshops as set up in the model literacy classrooms. 

Teachers in target classrooms collect sample pieces of writing for each student on at least 
three occasions during the year and analyze them for growth. 

Students in target classrooms are matched to text level. 


0.6907 


Factor 4: Use of data for planning and instruction 

Teachers in target classrooms evaluate student work in English/language arts using the 
New Standards Performance Standards for English/language arts and the primary literacy 
standards. 

Results on the previous year’s state assessments are analyzed to reveal strengths and 
weaknesses, to guide instruction, and to plan for results in the current year. 

Teachers in target classrooms use the America’s Choice performance standards and the 
elements to provide feedback to students and to guide instruction. 

Teachers in target classrooms take Running Records on each student on at least three 
occasions during the year and analyze them for growth. 

The Leadership Team uses “Planning for Results” in creating the School Plan. 

The Leadership Team reviews results from the Diagnostic and Assessment Tool to 
evaluate progress in implementing the design. 


0.7712 


Factor 5: Parent and community involvement 

The School Plan is endorsed by stakeholders (district, staff, parents, and partners). 

The Parent/Community Outreach Coordinator communicates with parents regarding the 
changes that are being implemented in the school and the reasons for the changes. 

There is a high level of involvement of parents/guardians in the 25-Book Campaign and in 
reading to their children. 

Teachers use the Home-School Notebooks to communicate with parents/guardians on a 
regular basis. 

The local community and school partners are aware and supportive of the standards- 
based reforms that the school is implementing. 


0.7952 



The local community and school partners provide tangible support to the school in 
increasing the number of books in classrooms. 
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Table Al. Factors, Items, and Estimates of Internal Consistency 
for the Elementary School Implementation Survey — Primary (continued) 

Cronbach’s 

Elementary Scales and Items Alpha 



Factor 6: Commitment to the design 

There is a high ievei of commitment to the America’s Choice design among staff in target 0.8835 
primary ciassrooms. 

There is a high ievei of commitment to the America’s Choice design among staff in target 
upper-eiementary ciassrooms. 

There is a high ievei of commitment to the America’s Choice design among aii staff. 

There is a high ievei of commitment to the America’s Choice design among students. 

There is a high ievei of commitment to the America’s Choice design among 
parents/guardians and the iocai community. 

Factor 7: Active and effective leadership 0.8745 

The Principal takes the lead role in advocating and impiementing the design. 

The Principai visits ciassrooms daiiy, inspecting the standard of student work and taiking 
to students about the quaiity of their work. 

The Principai coaches and mentors staff, providing feedback on ciassroom visits and on 
her/his anaiysis of student performance. 

The Leadership Team is estabiished, inciuding the Principai, Assistant Principal, Primary 
Coach, Upper-Eiementary Coach, and Parent/Community Outreach Coordinator. 

The Leadership Team meets reguiariy with set agendas that focus on impiementation of 
the design. 

There is effective distributed ieadership at this schooi in initiating and impiementing the 
America’s Choice design. 
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Table A2. Factors, Items, and Estimates of Internal Consistency for the 
Elementary School Implementation Survey — Upper Elementary 

Cron bach’s 

Elementary Scales and Items Alpha 

Factor 1: Understanding of the America’s Choice design 0.8575 

Teachers and administrators understand the connections between state/district content 
standards and the New Standards Performance Standards. 

There is a high ievei of understanding of the America’s Choice design among staff in 
target upper-eiementary ciassrooms. 

There is a high ievei of understanding of the America’s Choice design among aii staff at 
this schooi. 

There is a high ievei of understanding of the America’s Choice design among students. 

There is a high ievei of understanding of the America’s Choice design among 
parents/guardians and the iocai community. 

Factor 2: Implementation of school-level structural elements 0.7165 

The whole school community impiements the 25-Book Campaign. 

A Principai-sponsored Book-of-the-Month program is in piace. 

Exhibits of student work that approaches or meets the standards, aiong with appropriate 
standards and eiements, are dispiayed on buiietin boards in ciassrooms and haiis. 

Modei iiteracy ciassrooms are estabiished at the second and fourth grades and Skiiis 
Biock in a kindergarten/first-grade ciassroom. 

The Upper-Eiementary Coach impiements smaii group and tutoriai programs for students 
needing assistance with print/sound code. 

Summer iiteracy catch-ups are pianned for students who are significantiy beiow standard 
in iiteracy. 

Time is scheduied for meetings of teams of grade-ievei and same-subject teachers. 

The master scheduie inciudes a 2 V 2 -hour Literacy Biock and a 1-hour Mathematics Biock, 
and time is scheduied for meetings of grade-ievei and same-subject teachers. 

Ciass teachers are identified who wiii teach the same students for the next 2-3 years and 
aiiocated to teaching teams for the commencement of the foiiowing schooi year. 

There are scheduied Study Group meetings of teachers to read and discuss monographs. 

There are scheduied Teacher Meetings, coordinated by the Coaches, focusing on 
ciassroom teaching and iearning. 
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Table A2. Factors, Items, and Estimates of Internal Consistency for the 
Elementary School Implementation Survey — Upper Eiementary (continued) 

Cron bach’s 

Eiementary Scaies and Items Alpha 

0.7227 

Factor 3: Implementation of classroom-level structural elements 

Teachers in target ciassrooms (10 per coach) use the New Standards Performance 
Standards for Engiish/ianguage arts and the primary iiteracy standards to pian instruction 
and to deveiop iessons. 

Teachers in target ciassrooms impiement one Genre Study at each grade ievei. 



Teachers in target upper-eiementary ciassrooms (grades 3-5) impiement the Readers and 
Writers Workshops as set up in the modei literacy ciassrooms. 

Teachers in target ciassrooms coiiect sampie pieces of writing for each student on at ieast 
three occasions during the year and anaiyze them for growth. 

Students in target ciassrooms are matched to text ievei. 

Factor 4: Use of data for planning and Instruction 

Teachers in target ciassrooms evaiuate student work in Engiish/ianguage arts using the 
New Standards Performance Standards for Engiish/ianguage arts and the primary iiteracy 
standards. 

Resuits on the previous year’s state assessments are anaiyzed to reveai strengths and 
weaknesses, to guide instruction, and to pian for resuits in the current year. 

Teachers in target ciassrooms use the America’s Choice performance standards and the 
eiements to provide feedback to students and to guide instruction. 

Teachers in target ciassrooms take Running Records on each student on at ieast three 
occasions during the year and anaiyze them for growth. 

The Leadership Team uses “Pianning for Resuits” in creating the Schooi Pian. 



0.7712 



The Leadership Team reviews resuits from the Diagnostic and Assessment Tooi to 
evaiuate progress in impiementing the design. 



Factor 5: Parent and community involvement 

The School Plan is endorsed by stakeholders (district, staff, parents, and partners). 0.7952 

The Parent/Community Outreach Coordinator communicates with parents regarding the 
changes that are being implemented in the school and the reasons for the changes. 

There is a high level of involvement of parents/guardians in the 25-Book Campaign and in 
reading to their children. 

Teachers use the Home-School Notebooks to communicate with parents/guardians on a 
regular basis. 

The local community and school partners are aware and supportive of the standards- 
based reforms that the school is implementing. 

The local community and school partners provide tangible support to the school in 
increasing the number of books in classrooms. 
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Table A2. Factors, Items, and Estimates of Internal Consistency for the 
Elementary School Implementation Survey — Upper Eiementary (continued) 

Cronbach’s 

Eiementary Scaies and Items Alpha 



Factor 6: Commitment to the design 0.8835 

There is a high ievei of commitment to the America’s Choice design among staff in target 
primary ciassrooms. 

There is a high ievei of commitment to the America’s Choice design among staff in target 
upper-eiementary ciassrooms. 

There is a high ievei of commitment to the America’s Choice design among aii staff. 

There is a high ievei of commitment to the America’s Choice design among students. 

There is a high ievei of commitment to the America’s Choice design among 
parents/guardians and the iocai community. 

Factor 7: Active and effective leadership 0.8745 

The Principal takes the lead role in advocating and impiementing the design. 

The Principai visits ciassrooms daiiy, inspecting the standard of student work and taiking 
to students about the quaiity of their work. 

The Principai coaches and mentors staff, providing feedback on ciassroom visits and on 
her/his anaiysis of student performance. 

The Leadership Team is estabiished, inciuding the Principai, Assistant Principal, Primary 
Coach, Upper-Eiementary Coach, and Parent/Community Outreach Coordinator. 

The Leadership Team meets reguiariy with set agendas that focus on impiementation of 
the design. 

There is effective distributed ieadership at this schooi in initiating and impiementing the 
America’s Choice design. 
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Table A3. Factors, Items, and Estimates of Internal Consistency for the 
Middle School Implementation Survey 



Middle School Scales and Items Cronbach’s Alpha 

Factor 1: Understanding of the America’s Choice design 0.8624 

Teachers and administrators understand the connections between their state/district 
content standards and the New Standards Performance Standards. 

There is a high ievei of understanding of the America’s Choice design among Engiish 
teachers in target ciassrooms. 

There is a high ievei of understanding of the America’s Choice design among Math 
teachers in target ciassrooms. 

There is a high ievei of understanding of the America’s Choice design among aii staff 
at this schooi. 

There is a high ievei of understanding of the America’s Choice design among 
students. 

There is a high ievei of understanding of the America’s Choice design among 
parents/guardians and the iocai community. 

Factor 2: Implementation of school-level structural elements 0.7243 

The whole school community impiements the 25-Book Campaign. 

A Book-of-the-Month program is impiemented by the Principai and the Leadership 
Team. 

Exhibits of student work that approaches or meets the standards, aiong with 
appropriate standards and eiements, are dispiayed on bulietin boards in ciassrooms 
and haiis. 

Modei iiteracy and math ciassroom is estabiished at the eighth grade. 

Summer iiteracy and math catch-ups are pianned for students who are significantiy 
beiow standard. 

A pian is in piace to ensure aii ciassrooms have an adequate suppiy of ieveied books 
for the next schooi year. 

Ciass teachers are identified who wiii teach the same students for the next 2-3 years 
and aiiocated to teaching teams for the commencement of the foiiowing year. 

There are scheduied Study Group meetings of teachers to read and discuss 
monographs. 

There are scheduied Teacher Meetings, coordinated by the Coaches, focusing on 
ciassroom teaching and iearning. 



The Impact of America’s Choice on Writing Performance in Georgia: First-Year Resuits 



28 



Table A3. Factors, Items, and Estimates of Internal Consistency for the 
Middle School Implementation Survey (continued) 

Middle School Scales and Items Cronbach’s Alpha 



Factor 3: Implementation of classroom-level structural elements 0.6769 

English teachers in target classrooms implement Genre Studies at the seventh- and 
eighth-grade levels. 

English teachers in target seventh- and eighth-grade classrooms implement the 
Writers and Readers Workshops as set up in the model classroom. 

Math teachers in target classrooms implement the rituals and routines and use of 
standards-based assignments as set up in the model classroom. 

English teachers in target classrooms collect sample pieces of writing for each 
student on at least three occasions during the year and analyze them for growth. 

Factor 4: Use of data for planning and instruction 0.7800 

English and Math teachers in target classrooms evaluate student work using the New 
Standards Performance Standards and use the standards to guide the development 
of instruction. 

Other subject area teachers identify subject area standards to use in planning for 
instruction and analyzing student work. 

Results on the previous year’s state/district assessments are analyzed to reveal 
strengths and weaknesses, to guide instruction, and to plan for results in the current 
year. 

Results on the current year’s state/district assessments are analyzed to review 
progress, set targets for individual classes and students, and plan for results in the 
following year. 

Teachers in target classrooms use the America’s Choice performance standards and 
the elements to provide feedback to students and to guide instruction. 

The Leadership Team uses “Planning for Results” in creating the School Plan. 

The Leadership Team reviews results from the Diagnostic and Assessment Tool to 
evaluate progress in implementing the design. 

Factor 5: Parent and community involvement 0.7497 

The School Plan is endorsed by stakeholders (district, staff, parents, and partners). 

The Leadership Team communicates with parents regarding the changes that are 
being implemented in the school and the reasons for the changes. 

There is a high level of involvement of parents/guardians in the 25-Book Campaign. 

Teachers use the Home-School Notebooks to communicate with parents/guardians 
on a regular basis. 

The local community and school partners are aware and supportive of the standards- 
based reforms that the school is implementing. 

The local community and school partners provide tangible support to the school in 
increasing the number of books in classrooms. 
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Table A3. Factors, Items, and Estimates of Internal Consistency for the 
Middle School Implementation Survey (continued) 

Middle School Scales and Items Cronbach’s Alpha 



Factor 6: Commitment to the design 0.7813 

There is a high ievei of commitment to the America’s Choice design among Engiish 
teachers in target ciassrooms. 

There is a high ievei of commitment to the America’s Choice design among Math 
teachers in target ciassrooms. 

There is a high ievei of commitment to the America’s Choice design among aii staff at 
this schooi. 

There is a high ievei of commitment to the America’s Choice design among students. 

There is a high ievei of commitment to the America’s Choice design among 
parents/guardians and the iocai community. 

Factor 7: Active and effective leadership 0.7336 

The Principal and Assistant Principai take the iead roie in advocating and 
impiementing the design. 

The Principai and Assistant Principai visit ciassrooms daiiy, inspecting the standard of 
student work and taiking to students about the quaiity of their work. 

The Principai and Assistant Principai coach and mentor staff, providing feedback on 
their ciassroom visits and on their anaiysis of student performance. 

The Leadership Team is estabiished, inciuding the Principai, Assistant Principai (as 
Design Coach), Literacy Coach, Math Coach, and Parent/Community Outreach 
Coordinator. 

The Leadership Team meets reguiariy with set agendas that focus on impiementation 
of the design. 

There is effective distributed ieadership at this schooi in initiating and impiementing 
the America’s Choice design. 
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Appendix B: Results from Statistical Models of Impact 



Table Bl. Regression Results of the Model Predicting Baseline (2001) Writing Scores 
and Change in Writing Scores (2001 to 2002) for the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseiine Writing Scores in 2001 








Intercept 


-2.49*** 


0.02 


0.08 


School Size (per 100 students) 


0.01 


0.01 


1.01 


Percent Minority 


0.26** 


0.08 


1.30 


Percent Retained in Grade 


1.66** 


0.60 


5.26 


Percent Free/Reduced-Price Lunch Eligibie 


-1.97*** 


0.10 


0.14 


Percent Limited Engiish Proficient 


-0.20 


0.27 


0.82 


Percent Speciai Education 


-1 41*** 


0.37 


0.24 


Average Teacher Saiary (in thousands of doiiars) 


0.04*** 


0.01 


1.04 


Average Teacher Years of Experience 


-0.01* 


0.01 


0.99 


Pupii/Teacher Ratio 


-0.01 


0.01 


0.99 


Baseline Difference for Georgia’s Choice 


-0.03 


0.05 


0.97 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.38*** 


0.02 


1.46 


School Size (per 100 students) 


-0.01 


0.01 


0.99 


Percent Minority 


-0.38*** 


0.10 


0.68 


Percent Retained in Grade 


-0.44 


0.75 


0.64 


Percent Free/Reduced-Price Lunch Eiigibie 


1.05*** 


0.14 


2.86 


Percent Limited Engiish Proficient 


-0.12 


0.33 


0.89 


Percent Speciai Education 


0.07 


0.46 


1.07 


Average Teacher Saiary (in thousands of doiiars) 


-0.02~ 


0.00 


0.98 


Average Teacher Years of Experience 


-0.01 


0.01 


0.99 


Pupii/Teacher Ratio 


0.01 


0.01 


1.01 


Average Fifth-Grade Writing Levei for 2000-2001 


0.88*** 


0.06 


2.41 


Difference in Gains for Georgia’s Choice 


0.16* 


0.06 


1.17 


Thresholds 








5s 


1.89*** 


0.01 




54 


3.89*** 


0.01 




53 


5.92*** 


0.01 




52 


7.39*** 


0.03 






Variance 






Random Effect 


Component 






Baseline Writing Scores (2001) 


0.18*** 






Change in Writing Scores (2001-2002) 


0.26*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table B2. Regression Results of the Model Predicting Baseline (2001) Writing Scores 
and Change in Writing Scores (2001 to 2002) for the Eighth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-2.39*** 


0.02 


0.09 


School Size (per 100 students) 


0.01 


0.01 


1.01 


Percent Minority 


0.18 


0.13 


1.20 


Percent Retained in Grade 


-0.91 


0.88 


0.40 


Percent Free/Reduced-Price Lunch Eligible 


-2.39*** 


0.18 


0.09 


Percent Limited English Proficient 


-1.45~ 


0.79 


0.23 


Percent Special Education 


-1.91** 


0.70 


0.15 


Average Teacher Salary (in thousands of dollars) 


0.03* 


0.01 


1.03 


Average Teacher Years of Experience 


-0.01 


0.01 


0.99 


Pupil/Teacher Ratio 


-0.04** 


0.01 


0.96 


Baseline Difference for Georgia’s Choice 


-0.17* 


0.08 


0.84 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.13*** 


0.18 


1.14 


School Size (per 100 students) 


-0.01* 


0.01 


0.99 


Percent Minority 


-0.02 


0.10 


0.98 


Percent Retained in Grade 


-1.74* 


0.67 


0.18 


Percent Free/Reduced-Price Lunch Eligible 


-0.36* 


0.16 


0.70 


Percent Limited English Proficient 


-0.78 


0.58 


0.46 


Percent Special Education 


-0.52 


0.56 


0.59 


Average Teacher Salary (in thousands of dollars) 


0.01 


0.01 


1.01 


Average Teacher Years of Experience 


-0.01 


0.01 


0.99 


Pupil/Teacher Ratio 


-0.04** 


0.01 


0.96 


Average Eighth-Grade Writing Level for 2000-2001 


-0.94*** 


0.17 


0.39 


Difference in Gains for Georgia’s Choice 


0.17** 


0.06 


1.19 


Threshold 








52 


4 j_j_*** 


0.01 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.17*** 






Change in Writing Scores (2001-2002) 


0.06*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Appendix C: Results from Statistical Models Linking 
Implementation Measures and Impact 



Table Cl. Regression Results for Implementation of 
“Classroom-Level Structural Elements” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3 24*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.11 


0.25 


0.90 


Percent Retained in Grade 


0.74 


1.50 


2.10 


Percent Free/Reduced-Price Lunch Eiigibie 


-0.86* 


0.43 


0.42 


Percent Limited Engiish Proficient 


-0.87 


0.58 


0.42 


Percent Speciai Education 


-2.26* 


1.05 


0.10 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.03 


0.03 


1.03 


Pupii/Teacher Ratio 


-0.00 


0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.54*** 


0.05 


1.72 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.51- 


0.27 


0.60 


Percent Retained in Grade 


-3.07- 


1.71 


0.05 


Percent Free/Reduced-Price Lunch Eiigibie 


0.05 


0.47 


1.05 


Percent Limited Engiish Proficient 


-0.73 


0.64 


0.48 


Percent Speciai Education 


-3.08* 


1.20 


0.05 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.01 


0.03 


1.01 


Pupii/Teacher Ratio 


0.07*** 


0.02 


1.07 


Average Fifth-Grade Writing Levei for 2000-2001 


-0.95*** 


0.24 


0.39 


Classroom-Level Structural Elements — Grade 5 


0.02 


0.05 


1.02 


Thresholds 








5s 


2.03*** 


0.04 




54 


4 j_j_*** 


0.06 




5a 


6.09*** 


0.09 




52 


7.50*** 


0.14 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.35*** 






Change in Writing Scores (2001-2002) 


0.34*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C2. Regression Results for Implementation of 
“Classroom-Level Structural Elements” in the Eighth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 


Intercept 


-3.22*** 


0.06 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.80* 


0.32 


0.45 


Percent Retained in Grade 


-1.50 


1.59 


0.22 


Percent Free/Reduced-Price Lunch Eligibie 


-1.51* 


0.69 


0.22 


Percent Limited Engiish Proficient 


-0.04 


1.53 


0.96 


Percent Speciai Education 


-0.67 


1.68 


0.51 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupii/Teacher Ratio 


0.06 


0.04 


1.06 


Change in Writing Scores from 2001 to 2002 


Intercept 


0.30*** 


0.07 


1.35 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


0.80 


0.48 


2.23 


Percent Retained in Grade 


-0.56 


2.28 


0.57 


Percent Free/Reduced-Price Lunch Eligibie 


-0.00 


1.02 


1.00 


Percent Limited Engiish Proficient 


0.57 


2.23 


1.77 


Percent Speciai Education 


-0.01 


2.42 


0.99 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.04 


0.99 


Pupii/Teacher Ratio 


-0.09 


0.06 


0.91 


Average Eighth-Grade Writing Levei for 2000-2001 


2.12** 


0.71 


8.33 


Classroom-Level Structural Elements — Grade 8 


-0.04 


0.08 


0.96 


Threshold 


52 


4.18*** 


0.04 





Random Effect 


Variance 

Component 


Baseline Writing Scores (2001) 


0.07*** 


Change in Writing Scores (2001-2002) 


0.13*** 



~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C3. Regression Results for Implementation of 
“Commitment to the Design” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.25*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.12 


0.25 


0.89 


Percent Retained in Grade 


0.74 


1.52 


2.10 


Percent Free/Reduced-Price Lunch Eiigibie 


-0.87* 


0.43 


0.42 


Percent Limited Engiish Proficient 


-0.86 


0.58 


0.42 


Percent Speciai Education 


-2.27* 


1.05 


0.10 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.03 


0.03 


1.03 


Pupii/Teacher Ratio 


-0.00 


0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.54*** 


0.05 


1.72 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.51- 


0.27 


0.60 


Percent Retained in Grade 


-3.01- 


1.71 


0.05 


Percent Free/Reduced-Price Lunch Eiigibie 


0.14 


0.48 


1.15 


Percent Limited Engiish Proficient 


-0.66 


0.64 


0.52 


Percent Speciai Education 


-2.85- 


1.21 


0.06 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.01 


0.03 


1.01 


Pupii/Teacher Ratio 


0.07*** 


0.02 


1.07 


Average Fifth-Grade Writing Levei for 2000-2001 


-0.81*** 


0.25 


0.44 


Commitment to the Design — Grade 5 


0.00 


0.05 


1.00 


Thresholds 








5s 


2.04*** 


0.03 




54 


4 


0.04 




53 


6.09*** 


0.05 




52 


7.51*** 


0.08 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.13*** 






Change in Writing Scores (2001-2002) 


0.12*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C4. Regression Results for Implementation of 
“Commitment to the Design” in the Eighth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.22*** 


0.06 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.80- 


0.32 


0.45 


Percent Retained in Grade 


-1.50 


1.59 


0.22 


Percent Free/Reduced-Price Lunch Eligibie 


-1.52* 


0.69 


0.22 


Percent Limited Engiish Proficient 


-0.02 


1.53 


0.98 


Percent Speciai Education 


-0.66 


1.68 


0.52 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupii/Teacher Ratio 


0.06 


0.04 


1.06 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.31*** 


0.07 


1.36 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


0.89- 


0.49 


2.44 


Percent Retained in Grade 


-0.49 


2.35 


0.61 


Percent Free/Reduced-Price Lunch Eligibie 


0.12 


1.04 


1.13 


Percent Limited Engiish Proficient 


0.85 


2.23 


2.34 


Percent Speciai Education 


0.68 


2.45 


1.97 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.05 


0.99 


Pupii/Teacher Ratio 


-0.07 


0.06 


0.93 


Average Eighth-Grade Writing Levei for 2000-2001 


2.32** 


0.74 


10.18 


Commitment to the Design — Grade 8 


0.00 


0.06 


1.00 


Threshold 








52 


4.18*** 


0.04 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.67*** 






Change in Writing Scores (2001-2002) 


0.15*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C5. Regression Results for Implementation of 
“Use of Data for Planning and Instruction” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.24*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.12 


0.24 


0.89 


Percent Retained in Grade 


0.71 


1.50 


2.03 


Percent Free/Reduced-Price Lunch Eligibie 


-0.86* 


0.43 


0.42 


Percent Limited Engiish Proficient 


-0.87 


0.57 


0.42 


Percent Speciai Education 


-2.27* 


1.05 


0.10 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.03 


0.03 


1.03 


Pupii/Teacher Ratio 


-0.00 


0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.54*** 


0.05 


1.72 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.50- 


0.27 


0.61 


Percent Retained in Grade 


-2.91- 


1.72 


0.05 


Percent Free/Reduced-Price Lunch Eiigibie 


0.16 


0.47 


1.17 


Percent Limited Engiish Proficient 


-0.62 


0.64 


0.54 


Percent Speciai Education 


-2.76* 


1.2 


0.06 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.01 


0.03 


1.01 


Pupii/Teacher Ratio 


0.07*** 


0.02 


1.07 


Average Fifth-Grade Writing Levei for 2000-2001 


-0.76** 


0.24 


0.47 


Use of Data for Planning/lnstruction — Grade 5 


-0.01 


0.05 


0.99 


Threshoids 








5s 


2.03*** 


0.03 




54 


4 j_j_*** 


0.04 




5a 


6.09*** 


0.05 




52 


7.50*** 


0.08 




Random Effect 


Variance 

Component 






Baseiine Writing Scores (2001) 


0.12*** 






Change in Writing Scores (2001-2002) 


0.11*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C6. Regression Results for Implementation of 
“Use of Data for Planning and Instruction” in the Eighth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.22*** 


0.06 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.80* 


0.32 


0.45 


Percent Retained in Grade 


-1.50 


1.59 


0.22 


Percent Free/Reduced-Price Lunch Eligibie 


-1.53* 


0.69 


0.22 


Percent Limited Engiish Proficient 


-0.01 


1.53 


0.99 


Percent Speciai Education 


-0.65 


1.68 


0.52 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupii/Teacher Ratio 


0.06 


0.04 


1.06 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.30*** 


0.07 


1.35 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


1.02* 


0.47 


2.77 


Percent Retained in Grade 


-0.67 


2.28 


0.51 


Percent Free/Reduced-Price Lunch Eligibie 


-0.03 


1.01 


0.97 


Percent Limited Engiish Proficient 


1.31 


2.17 


3.71 


Percent Speciai Education 


0.67 


2.38 


1.95 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.00 


0.04 


1.00 


Pupii/Teacher Ratio 


-0.04 


0.06 


0.96 


Average Eighth-Grade Writing Levei for 2000-2001 


2.16** 


0.66 


8.67 


Use of Data for Planning/lnstruction — Grade 8 


0.12* 


0.05 


1.13 


Threshoid 








52 


4.18*** 


0.04 




Random Effect 


Variance 

Component 






Baseiine Writing Scores (2001) 


0.07*** 






Change in Writing Scores (2001-2002) 


0.13*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C7. Regression Results for Implementation of 
“Active and Effective Leadership” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.24*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.10 


0.25 


0.90 


Percent Retained in Grade 


-0.80 


1.51 


0.45 


Percent Free/Reduced-Price Lunch Eligibie 


-0.87* 


0.43 


0.42 


Percent Limited Engiish Proficient 


-0.87 


0.58 


0.42 


Percent Speciai Education 


-2.32* 


1.04 


0.10 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.03 


0.03 


1.03 


Pupii/Teacher Ratio 


-0.00 


0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.53*** 


0.05 


1.70 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.57- 


0.30 


0.57 


Percent Retained in Grade 


-2.47 


1.86 


0.08 


Percent Free/Reduced-Price Lunch Eiigibie 


-0.47 


0.53 


0.63 


Percent Limited Engiish Proficient 


-1.26- 


0.72 


0.28 


Percent Speciai Education 


-4.56*** 


1.32 


0.01 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.04 


0.03 


1.04 


Pupii/Teacher Ratio 


0.07** 


0.02 


1.07 


Average Fifth-Grade Writing Levei for 2000-2001 


-2.07*** 


0.24 


0.13 


Active and Effective Leadership — Grade 5 


0.06 


0.05 


1.06 


Thresholds 








5s 


2.03*** 


0.03 




54 


4 j_j_*** 


0.04 




53 


6.09*** 


0.05 




52 


7.50*** 


0.08 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.13*** 






Change in Writing Scores (2001-2002) 


0.17*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C8. Regression Results for Implementation of 
“Active and Effective Leadership” in the Eighth Grade 







Standard 


Odds 


Fixed Effect 


Estimate 


Error 


Ratio 


Baseiine Writing Scores in 2001 








Intercept 


-3.22*** 


0.57 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.79* 


0.32 


0.45 


Percent Retained in Grade 


-1.49 


1.59 


0.23 


Percent Free/Reduced-Price Lunch Eiigibie 


-1.55 


0.69 


0.21 


Percent Limited Engiish Proficient 


0.07 


1.53 


1.07 


Percent Speciai Education 


-0.60 


1.68 


0.55 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupii/Teacher Ratio 


0.05 


0.04 


1.05 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.30*** 


0.08 


1.35 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


1.08- 


0.53 


2.94 


Percent Retained in Grade 


-0.87 


2.64 


0.42 


Percent Free/Reduced-Price Lunch Eiigibie 


0.25 


1.14 


1.28 


Percent Limited Engiish Proficient 


1.23 


2.49 


3.42 


Percent Speciai Education 


1.04 


2.72 


2.83 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.00 


0.05 


1.00 


Pupii/Teacher Ratio 


-0.06 


0.06 


0.94 


Average Eighth-Grade Writing Levei for 2000-2001 


2.98*** 


0.71 


19.69 


Active and Effective Leadership — Grade 8 


0.09 


0.06 


1.09 


Threshoid 








52 


4.18*** 


0.04 






Variance 






Random Effect 


Component 






Baseiine Writing Scores (2001) 


0.07*** 






Change in Writing Scores (2001-2002) 


0.19*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C9. Regression Results for Implementation of 
“Parent and Community Involvement” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.24*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.11 


0.25 


0.90 


Percent Retained in Grade 


0.73 


1.51 


2.08 


Percent Free/Reduced-Price Lunch Eiigibie 


-0.86* 


0.43 


0.42 


Percent Limited Engiish Proficient 


-0.87 


0.58 


0.42 


Percent Speciai Education 


-2.27* 


1.05 


0.10 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.03 


0.03 


1.03 


Pupii/Teacher Ratio 


-0.00 


-0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.54*** 


0.05 


1.72 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.51- 


0.27 


0.60 


Percent Retained in Grade 


-2.93- 


1.65 


0.05 


Percent Free/Reduced-Price Lunch Eiigibie 


0.08 


0.47 


1.08 


Percent Limited Engiish Proficient 


-0.70 


0.64 


0.50 


Percent Speciai Education 


-2.96* 


1.20 


0.05 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.01 


0.03 


1.01 


Pupii/Teacher Ratio 


0.07*** 


0.02 


1.07 


Average Fifth-Grade Writing Levei for 2000-2001 


-0.91*** 


0.24 


0.40 


Parent and Community Involvement — Grade 5 


0.01 


0.05 


1.01 


Thresholds 








5s 


2.03*** 


0.03 




54 


4 j_j_*** 


0.04 




53 


6.09*** 


0.05 




52 


7.50*** 


0.08 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.12*** 






Change in Writing Scores (2001-2002) 


0.11*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table CIO. Regression Results for Implementation of 
“Parent and Community Involvement” in the Eighth Grade 







Standard 


Odds 


Fixed Effect 


Estimate 


Error 


Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.21*** 


0.06 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.80* 


0.32 


0.45 


Percent Retained in Grade 


-1.50 


1.59 


0.22 


Percent Free/Reduced-Price Lunch Eligible 


-1.52* 


0.69 


0.22 


Percent Limited English Proficient 


-0.02 


1.53 


0.98 


Percent Special Education 


-0.66 


1.68 


0.52 


Average Teacher Salary (in thousands of dollars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupil/Teacher Ratio 


0.06 


0.04 


1.06 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.30*** 


0.07 


1.35 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


0.90- 


0.50 


2.46 


Percent Retained in Grade 


-0.48 


2.35 


0.62 


Percent Free/Reduced-Price Lunch Eligible 


0.14 


1.04 


1.15 


Percent Limited English Proficient 


0.85 


2.25 


2.34 


Percent Special Education 


0.73 


2.54 


2.08 


Average Teacher Salary (in thousands of dollars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.05 


0.99 


Pupil/Teacher Ratio 


-0.08 


0.06 


0.92 


Average Eighth-Grade Writing Level for 2000-2001 


2.37** 


0.72 


10.70 


Parent and Community Involvement — Grade 8 


0.00 


0.06 


1.00 


Threshold 








52 


4.18*** 


0.04 






Variance 






Random Effect 


Component 






Baseline Writing Scores (2001) 


0.07*** 






Change in Writing Scores (2001-2002) 


0.15*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table Cll. Regression Results for Implementation of 
“School-Level Structural Elements” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseiine Writing Scores in 2001 








Intercept 


-3.24*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.11 


0.25 


0.90 


Percent Retained in Grade 


0.73 


1.50 


2.08 


Percent Free/Reduced-Price Lunch Eiigibie 


-0.86* 


0.58 


0.42 


Percent Limited Engiish Proficient 


-0.87 


1.05 


0.42 


Percent Speciai Education 


-2.27* 


0.00 


0.10 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.03 


1.00 


Average Teacher Years of Experience 


0.03 


0.02 


1.03 


Pupii/Teacher Ratio 


-0.00 


0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.54*** 


0.05 


1.72 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.52~ 


0.27 


0.59 


Percent Retained in Grade 


-3.01~ 


1.67 


0.05 


Percent Free/Reduced-Price Lunch Eiigibie 


0.05 


0.47 


1.05 


Percent Limited Engiish Proficient 


-0.71 


0.34 


0.49 


Percent Speciai Education 


-3.08** 


1.20 


0.05 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.01 


0.03 


1.01 


Pupii/Teacher Ratio 


0.07*** 


0.02 


1.07 


Average Fifth-Grade Writing Levei for 2000-2001 


-0.97*** 


0.05 


0.38 


School-Level Structural Elements — Grade 5 


0.02 


0.05 


1.02 


Thresholds 








5s 


2.03*** 


0.03 




54 


4 j_j_*** 


0.04 




53 


6.09*** 


0.05 




52 


7.50*** 


0.08 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.12*** 






Change in Writing Scores (2001-2002) 


0.11*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C12. Regression Results for Implementation of 
“School-Level Structural Elements” in the Eighth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseiine Writing Scores in 2001 








Intercept 


-3.22*** 


0.06 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.80* 


0.32 


0.45 


Percent Retained in Grade 


-1.50 


1.59 


0.22 


Percent Free/Reduced-Price Lunch Eligibie 


-1.51* 


0.69 


0.22 


Percent Limited Engiish Proficient 


-0.05 


1.53 


0.95 


Percent Speciai Education 


-0.67 


1.68 


0.51 


Average Teacher Saiary (in thousands of doiiars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupii/Teacher Ratio 


0.06 


0.04 


1.06 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.31*** 


0.07 


1.36 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


0.82- 


0.48 


2.27 


Percent Retained in Grade 


-0.48 


2.28 


0.62 


Percent Free/Reduced-Price Lunch Eligibie 


0.08 


1.01 


1.08 


Percent Limited Engiish Proficient 


0.81 


2.16 


2.25 


Percent Speciai Education 


0.65 


2.38 


1.92 


Average Teacher Saiary (in thousands of doiiars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.04 


0.99 


Pupii/Teacher Ratio 


-0.08 


0.06 


0.92 


Average Eighth-Grade Writing Levei for 2000-2001 


2.10** 


0.06 


8.17 


School-Level Structural Elements — Grade 8 


-0.03 


0.06 


0.97 


Threshold 








52 


4.18*** 


0.04 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.07*** 






Change in Writing Scores (2001-2002) 


0.13*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C13. Regression Results for Implementation of 
“Understanding of the America’s Choice Design” in the Fifth Grade 



Fixed Effect 


Estimate 


Standard 

Error 


Odds 

Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.25*** 


0.05 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.12 


0.25 


0.89 


Percent Retained in Grade 


0.74 


1.52 


2.10 


Percent Free/Reduced-Price Lunch Eligible 


-0.87* 


0.43 


0.42 


Percent Limited English Proficient 


-0.86 


0.58 


0.42 


Percent Special Education 


-2.27* 


1.05 


0.10 


Average Teacher Salary (in thousands of dollars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.03 


0.03 


1.03 


Pupil/Teacher Ratio 


-0.00 


0.02 


1.00 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.54*** 


0.05 


1.72 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.51- 


0.27 


0.60 


Percent Retained in Grade 


-3.00- 


1.70 


0.05 


Percent Free/Reduced-Price Lunch Eligible 


0.15 


0.48 


1.16 


Percent Limited English Proficient 


-0.64 


0.65 


0.53 


Percent Special Education 


-2.80* 


1.21 


0.06 


Average Teacher Salary (in thousands of dollars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


0.01 


0.03 


1.01 


Pupil/Teacher Ratio 


0.07*** 


0.02 


1.07 


Average Fifth-Grade Writing Level for 2000-2001 


-0.77** 


0.24 


0.46 


Understanding of the America’s Choice Design — Grade 
5 


-0.00 


0.05 


1.00 


Thresholds 








5s 


2.04*** 


0.03 




54 


4 j_2**-* 


0.04 




53 


6.09*** 


0.05 




52 


7.51*** 


0.08 




Random Effect 


Variance 

Component 






Baseline Writing Scores (2001) 


0.13*** 






Change in Writing Scores (2001-2002) 


0.12*** 







~p<.10, *p<.05, **p<.01, ***p<.001 
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Table C14. Regression Results for Implementation of 
“Understanding of the America's Choice Design” in the Eighth Grade 







Standard 


Odds 


Fixed Effect 


Estimate 


Error 


Ratio 


Baseline Writing Scores in 2001 








Intercept 


-3.22*** 


0.06 


0.04 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


-0.80* 


0.32 


0.45 


Percent Retained in Grade 


-1.50 


1.59 


0.22 


Percent Free/Reduced-Price Lunch Eligible 


-1.52* 


0.69 


0.22 


Percent Limited English Proficient 


-0.03 


1.53 


0.97 


Percent Special Education 


-0.66 


1.68 


0.52 


Average Teacher Salary (in thousands of dollars) 


0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.03 


0.99 


Pupil/Teacher Ratio 


0.06 


0.04 


1.06 


Change in Writing Scores from 2001 to 2002 








Intercept 


0.30*** 


0.07 


1.35 


School Size (per 100 students) 


-0.00 


0.00 


1.00 


Percent Minority 


0.88- 


0.49 


2.41 


Percent Retained in Grade 


-0.42 


2.34 


0.66 


Percent Free/Reduced-Price Lunch Eligible 


0.19 


1.06 


1.21 


Percent Limited English Proficient 


0.71 


2.26 


2.03 


Percent Special Education 


0.67 


2.44 


1.95 


Average Teacher Salary (in thousands of dollars) 


-0.00 


0.00 


1.00 


Average Teacher Years of Experience 


-0.01 


0.05 


0.99 


Pupil/Teacher Ratio 


-0.08 


0.07 


0.92 


Average Eighth-Grade Writing Level for 2000-2001 


2.38** 


0.74 


10.80 


Understanding of the America’s Choice Design — Grade 
8 


-0.02 


0.07 


0.98 


Threshold 








52 


4.18*** 


0.04 






Variance 






Random Effect 


Component 






Baseline Writing Scores (2001) 


0.07*** 






Change in Writing Scores (2001-2002) 


0.14*** 







~p<.10, *p<.05, **p<.01, ***p<.001 



